Towards prediction of Sense of Presence in immersive audiovisual communications

Tremendous progress has been made in audiovisual technologies in the last decades. Consequently, new technologies quality measures evolve and trend to be more user-centric. This is the reason why the Quality of Experience (QoE) assessment is presently meaningful and challenging, especially for users typical experiences during multimedia content consumption. Such an evaluation is the aim of this paper. More specifically, the Sense of Presence (SoP) was explored in place of QoE as it is a factor influencing the QoE. This paper presents the conducted subjective test investigating typical and practical user experiences. This latter consists of presenting one-minute video stimuli to twenty subjects, on three different devices (iPhone, iPad and UHD screen). Annotated subjective scores were collected and physiological signals (EEG, ECG, and Respiration) were recorded during the conducted subjective test. The resulting multimodal dataset, aiming an alternative evaluation of human experience while consuming multimedia, is publicly available. Introduction Audiovisual communication technologies have made tremendous progress in the last few decades. As an example, television broadcast has evolved from standard resolution at 25 interlaced frames per second, in black and white (gray scaled), to colour, and more recently higher resolutions up to 8K. Capture and rendition of High Dynamic Range content with Wide Colour Gamut have become reality thanks to cameras with advanced sensors and optics, and monitors featuring brightness levels up to 4K nits, with increased frame rates by several fold, when compared to the past. Attempts have been made to capture and display three dimensional (stereoscopic as well as multi view) content with some success, and omnidirectional and immersive video technologies seem to be around the corner, as witnessed by recent prototypes and early products. Similar progress has been made in audio technologies, where mono has been replaced by stereo, surround and 3D in the last decades. Furthermore, the modes of consumptions of audiovisual content are changing, with a wide range of services offering new experiences thanks to innovations in how information is stored, and delivered to consumers and professionals, thanks to powerful and feature rich devices and infrastructures that enable mobile and immersive audiovisual communications. In parallel with this progress, attempts have been made to assess such new experiences in a reliable, accurate, and reproducible manner, not only to quantify them better, but also to optimize the entire processing chain of media from its creation to its consumption, for a targeted level of experience. This paper positions itself in this latter category and aims at defining assessment methodologies that can measure an important dimension in emerging immersive media, namely, to implicitly evaluate the Sense of Presence (SoP) felt by subjects experiencing immersive audiovisual content in different situations. This is achieved by building upon a past similar attempt by the authors of this paper [13], where the sense of presence in a variety of audiovisual experiences was measured as a function of the modality and the quality of audiovisual content, from lower quality and no audio, standard definition video, to higher quality surround audio and ultra high definition video. The SoP was evaluated both explicitly, by means of a questionnaire, as well as implicitly, by recording selected physiological signals such as EEG, ECG, and respiration. This prior work showed that there is a correlation between sense of presence as experienced by human subjects and their physiological signals. In this paper, the previous earlier attempt mentioned above is extended, while still relying on the same fundamental approach of finding the correlation between SoP as experienced and explicitly assessed from subjects, and implicit prediction of the latter by analyzing physiological signals recorded from the same subjects during the same experiences. The type of immersive audiovisual experiences under consideration was extended in order to include those possible in typical mobile devices (such as when watching a movie on a phone or a tablet) in addition to Ultra High Definition (UHD) TV viewing experiences. A special attention was to scope out the typical and practical nature of users multimedia consumption. In addition to the analysis of the conducted experiment based on the explicit ratings, this paper offers a publicly available database in assessment of SoP in immersive communication1. Conducted experiments were based on the assumption that each device will induce to users different SoPs, referred to as Immersiveness Levels (ILs) in the remaining of this paper. Subjective ratings analysis showed that three immersiveness levels are observed. Nevertheless, the differences between the ILs produced by each device within used content is rather small. The reminder of the paper is structures as follows. In first Section, we derive the test from related work. We then describe the experiment design and run. The Section Subjective ratings analysis presents the results of the subjective tests regarding explicit assessments. In the last Section, a conclusion is given. 1htt p : //mmspg.ep f l.ch/SoPMD (a) C1 (b) C2 (c) C3 (d) C4 (e) C5 (f) C6 (g) C7 (h) C8 (i) C9 (j) C10 Figure 1: Frame example of each sequence: testing dataset ((a)-(i)) and testing dataset (j).